ai scraper
Reddit puts AI scrapers on notice
Reddit has a warning for AI companies and other scrapers: play by our rules or get blocked. The company said in an update that it plans to update its Robots Exclusion Protocol (robots.txt The company said it will also continue to block and rate-limit crawlers and other bots that don't have a prior agreement with the company. The changes, it said, shouldn't affect "good faith actors," like the Internet Archive and researchers. Reddit's notice comes shortly after multiple reports that Perplexity and other AI companies regularly bypass websites' robots.txt Perplexity's CEO, in a recent interview with Fast Company, said that the protocol is "not a legal framework."